N ov 2 00 4 Notes on Stein - Sahi representations and some problems of non L 2 harmonic analysis Neretin

We discuss one natural class of kernels on pseudo-Riemannian symmetric spaces. Recently, Oshima [67] published his formula for c-function for L 2 on pseudo-Riemannian symmetric spaces (see also works of Delorm [12] and van den Ban– Schlichtkrull [3], [4]). After this, there arises a natural question about other solvable problems of non-commutative harmonic analysis. In the Appendix to the paper [57], the author proposed a series of non L 2-inner products in spaces of functions on pseudo-Riemannian symmetric spaces and conjectured that this object is reasonable and admits an explicit harmonic analysis. In this work, we discuus the problem in more details, in particular, we obtain the Plancherel formula for these kernel for Riemannian symmetric spaces U(n), U(n)/O(n), U(2n)/Sp(n). We also give a new proof of Sahi's results [76]. 0. Introduction 0.1. Inner products defined by kernels. Starting the famous works of Bargmann [5] and Gelfand–Naimark [21], various inner products having the form f 1 , f 2 = G/H×G/H K(x, y)f 1 (x)f 2 (y)dx dy (0.1) are quite usual in the theory of unitary representations. Here G/H is a homogeneous space and K(x, y) is a distribution (a 'kernel') on G/H × G/H. The group G acts in a space of functions on G/H by transformations having the form ρ(g)f (x) = f (gx)γ(g, x), (0.2) where γ(g, x) is some function ('multiplier') on G × G/H. We intend to obtain a unitary irreducible representation; under this requirement, the kernel K(x, y) is uniquely determined by the explicit expression for the multiplier γ(g, x). An actual evaluation of the kernel is not difficult. K(x, y)f (x)f (y)dx dy has no visible reasons to be positive. Usually, positive definiteness of a given inner product of the form (0.1) is a nontrivial problem. Example. We consider group SU(1, 1) consisting of 2 times2 matrices a b b a , where |a 2 | − |b| 2 = 1. It acts in the space of functions 1